Dynamical Jahn-Teller Effect in Spin-Orbital Coupled System 
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Dynamical Jahn-Teller (DJT) effect in spin-orbital coupled system on a honeycomb lattice is examined, mo- 
tivated from recently observed spin-liquid behavior in Ba3CuSb209. Effective vibronic Hamiltonian, where the 
superexchange interaction and the DJT effect are taken into account, is derived. We find that the DJT effect 
induces a spin-orbital resonant state where local spin-singlet states and parallel orbital configurations are en- 
tangled with each other This spin-orbital resonant state is realized in between an orbital ordered state, where 
spin-singlet pairs are localized, and an antiferromagnetic ordered state. Based on the theoretical results, a pos- 
sible scenario for Ba3CuSb20g is proposed. 

PACS numbers: 75.25.Dk, 75.30.Et,75.47.Lx 



No signs for long-range magnetic ordering down to the low 
temperatures, terms the quantum spin-liquid (QSL) state, are 
one of the fascinating states of matter in modern condensed 
matter physics A number of efforts have been done to 
realize the QSL states theoretically and experimentally. One 
prototypical example is the well known one-dimensional an- 
tiferromagnetic spin chains in which large quantum fluctua- 
tion destroys the long-range spin order even at zero temper- 
ature and realizes a spin-singlet state without any symmetry 
breakings. Another candidate for the QSL states has been 
searched for a long time in frustrated magnets. An organic salt 
K-(BEDT-TTF)2Cu2(CN)3 wifli a triangular lattice H and 
an inorganic herbertsmithite ZnCu3(OH)6Cl2 with a Kagome 
lattice [30 are some examples. Several theoretical scenarios 
for realization of the QSL, such as Z2-spin liquid 101, spin- 
nematic state jst], spinon-deconfinement 0] and so on, have 
been proposed so far. 

A transition-metal oxide of our present interest is a new 
candidate of the QSL state, Ba3CuSb209, in which S = 1/2 
spins in Cu^+ ions are responsible for the magnetism 10, S . 
There are no signs of magnetic orderings down to a few hun- 
dred mK, in spite of the effective exchange interactions of 30- 
50K. Temperature dependences of the magnetic susceptibil- 
ity and the electronic specific heat are decomposed into the 
gapped component and the low-energy component, which is 
attributed to the so-called orphan spins |8i|. It was believed 
that the QSL behavior originates from the magnetic frustration 
in a Cu^+ triangular lattice. Recent detailed crystal-structural 
analyses reveal that Cu ions are regularly replaced by Sb ions, 
and form a short-range honeycomb lattice j^. One character- 
istic in the present QSL system is that there is the orbital de- 
gree of freedom; two-fold orbital degeneracy in Cu^+ is sug- 
gested by the three-fold rotational symmetry around a Cu^+. 
Almost isotropic g-factors observed in the electron-spin reso- 
nance (ESR) provide a possibility of no static long-range or- 
bital orders and novel roles of orbital on the QSL. 

In this Letter, motivated from the recent experiments in 
Ba3CuSb209, we examine a possibility of the QSL state in a 
honeycomb-lattice spin-orbital system. Beyond the previous 
theories for QSL in quantum magnets with the orbital degree 
of freedom 11 91 41311 . the present theory focuses on the dynam- 



ical Jahn-Teller (DJT) effect, which brings about a quantum 
tunneling between stable orbital-lattice states. This is feasible 
in the crystal lattice of Ba3CuSb209 where Og octahedra sur- 
rounding Cu^+ are separated with each other, unlike the per- 
ovskite lattice where nearest-neighboring (NN) two octahedra 
share an O^^. The spin-orbital superexchange (SE) interac- 
tions between the separated NN Cu^+ are comparable with 
the vibronic interactions, and a new state of matter is expected 
as a result of the competition between the on-site Jahn-Teller 
(JT) and inter-site SE interactions. We derive the low-energy 
electron-lattice coupled model where the SE interaction, the 
JT effect, and the lattice dynamics are taken into account. It is 
discovered that a spin-orbital resonant state, where the two de- 
grees of freedom are entangled with each other, is induced by 
the DJT effect. We examine connections of the present quan- 
tum resonant state to a spin ordered state and an orbital or- 
dered state, and provide a possible scenario for Ba3CuSb209. 

In a CuOg octahedral cage with the Csv symmetry, one 
hole in Cu^+ occupies doubly degenerate E orbitals, which 
are approximately Sd^^'^^r^ and 3c?^2„y2. We set up a model 
where the CuOg octahedra are arranged on a honeycomb lat- 
tice [see Fig. \lla)] by taking into account the recent x-ray 
scattering experiments fl]. The starting Hamiltonian consists 
of the electronic term Hdp and the electron-lattice interac- 
tion term He-i- The electronic term is the dp-type Hamilto- 
nian where the degenerate 3d322_^2 and 3(i2,2_y2 orbitals in 
Cu^+ and the three 2p orbitals in 0^~ are introduced. The 
electron transfers between the Cu and O sites (tdp), those 
between the different O sites (tpp), and the Coulomb and 
exchange interactions in Cu and O ions are taken into ac- 
count. Details are given in Supplemental Material (SM) ll4ll . 
As for the electron-lattice interaction term, the two vibra- 
tional modes for an Oe cage with the Eg symmetry, Qu and 
Qy, couple with the 3d3z2_,.2 and 3dx2_y2 orbitals. This is 
given as n.^i = EJ-(2M)-i + dVidQl,)) + 

Mu^py2 + 2A{TfQ,, + T^-Qui) + B{Ql^ - iQl.Qm)], 
here pi = V Qui + Qvi is an amplitude of the lattice distor- 
tion, and Ti is the pseudo-spin (PS) operator for the doubly- 
degenerate orbitals with an amplitude 1/2. The first and sec- 
ond terms describe the harmonic vibrations with frequency u 
and ionic mass M. The third term describes the JT coupling 
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FIG. 1: (color online), (a) A honeycomb lattice structure for 
Ba3CuSb20g. The axes of coordinates, x, y, z, are taken to be along 
the NN Cu-0 bond directions in an octahedron, (b) A contour map of 
the lower AP surface t/'^' on the Qu-Qv plane. A blue broken line 
shows a circle for p — po. Schematic Oe distortions at the potential 
minima and maxima are also shown, (c) A schematic picture for the 
spin-orbital resonance state. Shaded bonds represent the spin-singlet 
and parallel-orbital bonds. 



with a coupling constant A{> 0), and the last term represents 
the anharmonic lattice potential, where B is negative in most 
ofCu2+. 

Here, we compare several energy scales. In the electronic 
part, the on-site Coulomb interaction at Cu'^^ is Ud ~5-10eV 
which is much larger than the Cu-O and O-O hybridizations 
tpd ^ tpp ^ leV. The SE interaction is J ~'l-10meV, which 
is smaller than that in the high-Tc cuprates due to a large dis- 
tance between the NN Cu sites. As well known in the E ® e 



JT problem |15|], two adiabatic potentials (APs) are given by 
= A'/a;2p2±^p + Scos36l where6' = tan-i(Q„/0„). 
The largest energy scale in the vibronic part is provided by 
the JT energy Sjt[= A^/{2Muj'^) = Muj'^pl/2] which is 
of the order of 0.1-leV. We introduce po = A/{Muj'^) at 
which the lower AP, U^^\ takes its minima in the case of 
5 = 0. The anharmonic-potential energy i?AH(= I^IPq)' the 
lattice kinetic energy for the azimuthal rotation on the Qu-Qv 
plane E'j-ot [— / Ejt = 2/ (Mpg)], and the zero-point vibra- 
tional energy i?zoio(= ^/2) are of the orders of 1-IOmeV, 1- 
lOmeV and 10-lOOmeV, respectively. From these estimations, 
we draw a following picture for the low-energy electronic and 
vibrational states up to about lOmeV: the local vibronic sys- 
tem is confined on the lower AP surface, and a vibronic state 
dynamically tunnels during the three potential minima caused 
by the anharmonic potential [see Fig. [lib)]. This local vi- 
bronic state competes with the spin-orbital state stabilized by 
the inter-site SE interactions. 

Based on this picture, the low-energy effective model is de- 
rived. The SE interactions between the NN Cu ions are ob- 
tained by the perturbational calculations. The effective Hamil- 



tonian is given by 



JssrSi ■ Sj^T^ '^j) J SSTT Si ■ SjT^T, 



'-'ssTT^i ^j'i 'j ' 'Jssyy^i ^J-^i j 



(1) 



where Si is the spin operator with an amplitude 1/2, and 
{ij)i is the NN Cu sites along one of the bond directions 
l{= x,y,z) defined in Fig. [TJa) lfl6ll . For convenience, 
we introduce t' = cos(27rn//3)T/ — sm{2TTni/3)Tf and 
f/ = cos(27r7i,/3)T'f + sin(27rn//3)Tf with {n^,n^,ny) = 
(0, 1, 2). The eigen state of r' with the eigen value H-I/2 (- 
1/2) corresponds to the state where d^i2_^.2 (c?m2„„2) orbital 
is occupied by a hole and {I, m, n) are the cyclic permutations 
of the Cartesian coordinates. Detailed derivation is given in 
SM ll4ll . For the two NN Cu sites, we find that this exchange 
Hamiltonian favors an antiferromagnetic (AFM) and ferro- 
type orbital configuration in a wide parameter region which 
includes a realistic parameter set for Ba3CuSb20g lfl4l Wh . 
This is due to the Cu-O-O-Cu exchange paths, and is in con- 
trast to the conventional ferromagnetic and staggered-type or- 
bital configuration realized in the corner-sheared two octahe- 
drad. 

As for the vibrational term, we focus on the rotational mode 
on the lower AP surface where an angle 6 = tan^^((5i,/(3„) 
is varied with a fixed amplitude at p = po- The rotational- 
mode Hamiltonian for a single octahedron isgiven by T-L^ot = 
-(2Mpg)-iSV(96'2) + BplcosW H [ll. The second 
term takes its minima (maxima) at angles 0^,^ — Trii + 2iti'/3 
with ly = (0,1,2) for p = (/i = 1). The lowest-six 
eigen states for "Hrot are transformed into the Wannier-type 
wave functions (/)^,y(6') which are almost localized around 
Ofji, [see Fig. [Ttb)]. We adopt, as the basis set, the six vi- 
bronic wave functions $^^(r;6') ~ ^{r;9)(f>f^,y{9), where 
^{r;9) = ^^2_y2 (r) cos(6'/2) — ■i/;322_r2 (r) sin(6'/2) is the 
wave function for a hole. Then, the low-energy vibronic 
Hamiltonian is represented by a simple form; 
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i 1=0 



(2) 



where erf is the z component of the Pauli matrices and is de- 
fined for the index p=0 and 1. We introduce 3x3 matri- 
ces {D^)^yi ~ \£ii^v'\, where ei^^i are the Levi-Civita com- 
pletely antisymmetric tensors, for the index v. The first and 
second terms represent the vibronic motion and the anhar- 
monic lattice potential, respectively, and Jdjt and Jah are 
the constants of the order of cj^/i?jT and |i?|po, respectively. 
The condition Jd.it/^ah < 1/2 is required to reproduce the 
eigen states of 'Hrot, but we regards Jdjt/>/ah(= Jd) as a 
free parameter, for convenience. The lowest energy state for 
HjT is a doublet, which corresponds to the clockwise and 
counter-clockwise rotations described by the nodeless wave 
functions in the 6 space. The detailed derivation of ^jt is 
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FIG. 2: (color online), (a) Staggered spin moment A4s, and two PS 
moments Mt+ and A4t- ■ (b) Spin-correlations {Sa-Sb} and {Sb ■ 
Sc) (bold lines), and spin-dimer correlations A'ab:BC, A'ab:Cd and 
A'ab;DE (thin lines). The inset shows a 6-site cluster. We obtain 
{Sa ■ Sb) = {Sc ■ Sd) = {Se ■ Sy) and (5b ■ ^c) = {So ■ 
Se) = {Se ■ Sa)- (c) Spin-orbital correlation function G, and the 
three-fold orbital order parameter A^ps. The broken line indicates 
joc- (d) (e) Two kinds of the three-fold orbital ordered states, (f) A 
schematic spin-orbital resonance state. Arrows represent the orbital 
PSs. 



given in SM Ill4ll . Finally, the exchange and vibronic terms 
are taken together as the low-energy effective Hamiltonian; 

HcS = Hcxch + "HjT- 

This Hamiltonian is analyzed by the exact diagonalization 
(ED) method combined with the mean-field (MF) approxima- 
tion, and the quantum Monte-Carlo simulation (QMC) with 
the MF approximation, termed the EDh-MF and the QMCh-MF 
methods, respectively. We introduce mainly the results by the 
EDh-MF method. The mean-field type decouplings are intro- 
duced in the exchange interactions which act on the edge sites 
of clusters. The Hamiltonian for a 6-site cluster under the 
MFs is diagonalized by the Lanczos algorithm, and the MFs 
are determined consistently with the states inside the cluster 
The adopted parameter values are Jss/Jah = 0.15 and given 
in Ref . 111411 . We find that the obtained results do not depend 
qualitatively on the parameter J^s/Jah between 0.075 and 
1.5. The exchange Hamiltonian "Hcxch is also analyzed by the 
MF approximation and by the ED method as supplementary 
calculations. 

Spin and orbital structures are examined as functions of am- 
plitude of the DJT effect, i.e. ju ~ Jd.it / Jah- We introduce 
the staggered spin moment Ms = ^ X^il^l) '*^!"' ^^'^ '^^^ 
PS moments defined by A^t-+ = ~g(''A + ''b + '''c + '''d + 
t| + 4) and Mr- = -i(rl + ^ + ^ + + ri + t§) 
where subscripts A-F indicate sites in the cluster [see the in- 
set of Fig. |2b)]. We note that Mt+ and Mr- take their 
maxima of 0.5 in the three-fold orbital ordered states shown 



in Figs. |2d) and (e), respectively. A difference between the 
two, A^ps = {■Mr+ — Mr-), is regarded as an order pa- 
rameter for the three-fold orbital ordered state. The numer- 
ical results are plotted in Fig. I2a). There is a critical value 
of jo, termed ji3c(= 0.75). For jd <C joc. Mt+ = 0.5 
and M.T- = —0.25, implying a symmetry breaking, i.e. the 
three-fold orbital ordered state shown in Fig. |2d). This or- 
bital order is also suggested by the analyses of Hcxch- With 
increasing jo, absolute values of both the two PS moments are 
reduced. These reductions are reproduced by the QMCh-MF 
method. In a region of j^i > Jdc, {Air+) = (A^t-) which 
is interpreted as a superposition of the two PS configurations 
shown in Fig.|2f). 

As for the spin sector, a finite staggered spin moment 
[see Fig. |2a)], as well as a finite local spin moment {S^) 
at each site, do not appear in a whole parameter region of 
jo- As shown in Fig. |2|b), there are the two inequivalent 
NN spin correlations {Si ■ Sj) for ju < jdc- large val- 
ues are shown in the bonds where the PSs are parallel with 
each other. On the other hand, for jo > joc, {Si ■ Sj) for 
all bonds are equivalent. Spin-dimer correlations defined by 
Kij -ki = {{Si- Sj){Sk- Si)) provide information for the bond 
correlation. We plot A'ab:BCi -f^'AB:CD and Kab-.ue which 
represent the bond-correlation functions for the NN bonds, 
the 2nd-NN bonds, and the 3rd-NN bonds, respectively. In 
a whole parameter region, the 2nd-NN bond correlation func- 
tion, Kab-.cb, is the largest. It is also shown that A'ab:CD 
in jo < joc is larger than that in jo > joc- The results 
in jo < joc are interpreted as a valence-bond solid state, 
where spin-singlet pairs are localized at the bonds, in which 
the NN PSs are parallel with each other. This spin and orbital 
structure is also confirmed in the analysis by the QMC+MF 
method. Above joc, this state is changed into a superposi- 
tion of the PS configurations, where the spin-singlet dimers 
are not localized in specific bonds, suggested by a reduction 
of KxB-cT) and enhancements of Kab-.bg and Kxb-'D-e- 

We expect from these data that, above jdc, the local spin 
singlet and the parallel PS configuration are strongly entan- 
gled quantum mechanically. This is confirmed by the spin- 
orbital correlation function defined by G = ^'yl^ 
withG,^^- = l&[{{SvSj){TlT'j))-{Si-Sj){Tyj)]- The results 
are presented in Fig.|2|c). Spin and orbital sectors are decou- 
pled at ju = 0, and are strongly entangled near and above 
Jdc- This is consistent with the picture where the spin singlet 
and the parallel PS configuration are realized as a quantum 
mechanical superposition. 

To examine a connection of the newly finding spin-orbital 
resonant state with the well-known spin ordered state in the 
honeycomb-lattice Heisenberg model, we introduce the arti- 
ficial field acting on the orbital-lattice sector This Hamilto- 
nian is given by Hai = -Iim af (g) where {F),yi,i = 
X)/ ^ii^v is a 3 X 3 matrix. This field lifts the doublet 
ground state for Hjt, and reduces the low-energy effective 
model into the AFM Heisenberg model without the orbital de- 
gree of freedom. The phase diagram on a plane of jo and Km 
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FIG. 3: (color online), (a) Phase diagram on the plane of Jdjt / Jah 
and hM/JAH- The interaction between the octahedra, Winter, is 
taken into account in (b), where the parameter is chosen to be 
K/Jah = 0.1. 



is shown in Fig. [3 a). The Neel order appears for large hM- 
We find the spin-orbital resonant state is realized in between 
the spin ordered state and the orbital ordered state which is 
realized in small ]£> and h^j- 

The spin-orbital resonant state is realized in more realis- 
tic parameter space, when the lattice effect is taken into ac- 
count beyond the independent octahedra approximation. The 
interactions between the Og octahedra are modeled by the 
Hamiltonian: Hintcr = K/MJ:{^J)S^QLQJu-QLQrv) 
where (3-„ = cos(2n/7r/3)Qi„ + sin(2n/7r/3)Qi„ and Q-.^ 
— sin(2ni7r/3)QiM + cos(2n;7r/3)(5it;. This is derived from 
the elastic interaction between the two O ions belonging to the 
NN octahedra, and K is the spring constant. The numerical 
results for Hcs + "Hintor are presented in Fig.[3b). The spin- 
orbital resonant phase is shifted to the lower side of jo and 
hM- In particular, without the artificial field (hnj ~ 0), Jdc is 
decreased down to 0.35. This result satisfies the condition of 
jn < 0.5, in which Hjt is valid as an effective Hamiltonian 
for the low-energy vibronic states of Hmt- 

From the above analyses, it turns out that the present find- 
ing spin-orbital resonant state emerges under the quantum or- 
bital state, and a stability of this state is controlled by the DJT 
effect. We have found previously in Ref. |20(I that, in the 
honeycomb lattice orbital-only model, Horb = '''1''']^ 
conventional long-range orders, as well as any translational- 
symmetry broken states, are found neither in quantum cal- 
culations at zero temperature nor in finite-temperature clas- 
sical calculations. A suggested ground state is a resonant 
orbital state, i.e., a superposition of the states where a hon- 
eycomb lattice is covered by the NN bond dimers with the 
parallel PS configuration. Here, we connect the present ex- 
change Hamiltonian Hcxch in Eq. ([T]i to "Horb, and exam- 
ine the relation between the two resonant states. For this 
end, we generalize the electron transfer as tpd tpdiv) 
by introducing a parameter 77, and apply the staggered mag- 
netic field as 7ih = ~I^J2ii~^T^i- Detailed models are 
given in SM ll4ll . The present SE Hamiltonian Hcxch and 
the orbital-only model Horh correspond to (77, h) = {tt/3, 0) 
and (7r/2, 00), respectively, and the orbital states between the 
two limits are examined continuously. In Fig. IH a three- 
dimensional plot for the phase diagram as functions of h. 
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FIG. 4: (color online). Phase diagram for the model where the gener- 
alization of the electron transfer, 77, and the staggered magnetic field, 
h, are taken into account [see text]. The SE interaction model Hoxch 
and the orbital-only model "Horh correspond to (ry, /i) — (7r/3,0) 
and (7r/2, 00), respectively. A shaded area shows the spin-orbital 
resonant phase. 



T] and ju is presented. The spin-orbital resonant state at 
{i],h) = (7r/3,0) and jD > joc = 0.75 continuously con- 
nects to the orbital resonant state in the orbital-only model 
realized in (77, h) = {tt/2, ^ Jah)- This fact implies that the 
present spin-orbital state belongs to the same class of the or- 
bital resonance state in the orbital-only model, and supports 
the picture shown in Fig. [TJc). We note that a stability of 
the orbital resonant state in Horh was examined by using sev- 
eral calculation methods and was confirmed at the infinite size 
limit ll20ll . We strongly suppose that the spin-orbital resonant 
state also does not break translational-symmetries, and is sur- 
vived beyond the present finite-size calculations. 

Based on the theoretical calculations, we propose a possi- 
ble scenario for Ba3CuSb209 10, Si- The x-ray experimental 
results suggest a kind of short-range honeycomb-lattice do- 
main of the order of lOA, which justifies the present finite- 
size cluster analyses to mimic the realistic situation. The 
observed positive Weiss constant is not trivial in orbital de- 
generate magnets where the ferromagnetic interaction is usu- 
ally dominant, and can be explained by the present calculation 
where the exchange paths are properly taken into account in a 
realistic lattice structure. For the no long-range spin and or- 
bital orders in the hexagonal samples, the present spin-orbital 
resonant state is one plausible candidate. The temperature de- 
pendence of the magnetic susceptibility is decomposed into 
the Curie tail and a gapped component. When the former is 
attributed to the orphan spins, the latter is consistent with the 
present spin-orbital resonant state where the short-range spin 
singlets are realized. This interpretation is also applicable to 
the electronic specific heat; the peak around 20K, which is 
robust against the magnetic field, is associated with the spin- 
orbital resonant state, and the low temperature component is 
due to the orphan spins. The existence of the gapped magnetic 
excitation is also suggested by the nuclear magnetic resonance 
measurements 12111 . A detailed magnetic structure is revealed 
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by the inelastic neutron scattering; a finite excitation-energy 
spectrum, interpreted as a singlet-triplet excitation, shows a 
characteristic momentum dependence which does not indi- 
cate a simple local NN singlet pairs, but some spatial distri- 
bution. This might be explained in the present spin-orbital 
resonant state, where the spin-singlet pairs are not localized, 
but are shuffled quantum mechanically together with the or- 
bital configuration. The time scale for the spin-orbital motion 
in the spin-orbital resonant is governed by the local DJT ef- 
fect w^/E'jT ^1-lOmeV and the inter-site exchange interac- 
tion J '--^l-lOmeV, both of which are in between the ESR time 
scale 10~^s) and the x-ray time scale (~ 10~^^s). These 
relations are suitable for the experiments which seems to be 
contradicted with each other at a glance: the almost isotropic 
ESR signal and the anisotropic extended x-ray absorption fine 
structure (EXAFS) data. Further comparison between the the- 
ory and experiments are necessary to confirm the spin-orbital 
resonant picture. The vibronic excitations measured by Ra- 
man and x-ray scattering spectroscopies are possible methods 
to check the scenario. 

In summary, we find that the spin-orbital resonant state is 
realized by the DJT effect in a honeycomb lattice spin-orbital 
model. Rearrangements of the spin-singlet pairs and the paral- 
lel orbital PS configurations are strongly entangled with each 
other. From the viewpoint of the orbital ordered state, the 
DJT effect strengthens the orbital fluctuation, diminishes the 
local orbital moment, and then induces the spin orbital reso- 
nant state. The present study does not only provides system- 
atic explanations for the recent experiments in Ba3CuSb209, 
but also proposes a new route to the spin-liquid state in spin- 
orbital coupled systems where the dynamical JT effect is rel- 
evant. 
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Supplemental Material: Dynamical Jahn-Teller 
Effect in Spin-Orbital Coupled System 

In this Supplemental Material, detailed derivations of the su- 
perexchange Hamiltonian and the effective vibronic Hamilto- 
nian are presented. 



SUPEREXCHANGE HAMILTONIAN 

In this section, a derivation and explicit forms of the su- 
perexchange Hamiltonian in Eq. (1) in the main article are 
presented. 



Derivation of the superexchange Hamiltonian 

We consider a two-dimensional honeycomb lattice where a 
CuOg octahedron is located in each site as shown in Fig. 1(a) 
in the main article. An octahedron and a Cu site are labeled 
by an index i and their positions are denoted by r^. The O 
sites which belong to the i-th octahedra are labeled by the 
indexes i and ±x,±y,±z). Positions of these O sites 
are r; + ds, where ds is a vector along a direction 5 with an 
amplitude of the nearest neighbor (NN) Cu-O bond distance. 
The axes of coordinates, x, y, z, are taken to be along the NN 
Cu-O bond directions in an octahedron as shown in Fig. 1(a) 
in the main article. A NN Cu-Cu bond, where the exchange 
paths are located on the /m-plane (see Fig. |5]), is labeled by 
an index n, where {I, m, n) are the cyclic permutations of the 
Cartesian coordinates. 

We start from the dp-type Hamiltonian where the doubly- 
degenerate eg orbitals in Cu sites and the three 2p orbitals in 
O sites are introduced. The Hamiltonian is given by 



Ht + HA + n 



Ud 



n 



Up- 



(3) 




FIG. 5: 
plane. 



Schematic two-types of the exchange processes on the xy 



The first term represents the electron transfer between the Cu 
and O orbitals and that between the O orbitals: 

iS 7i;cr 

- E T.{C[^SjS'Plsn.P^S'n'. + H.c.), (4) 
where di^a is the annihilation operator for a Cu hole at the i-th 



octahedron with orbital 7(= 3z —i 



-y ) and spin o'(=t 



, I), and piSr-ia is the annihilation operator for an O hole at the 
(5-th O site in the i-th octahedron with orbital x, y, z) 
and spin a. A symbol < id;jd' > represents the NN O sites 
labeled by {i,6) and {j,6'), and tj'^.^g and t'^ligjg, are the 
corresponding transfer integrals. The second term in Eq. (O 
represents the energy differences given by 



n 



A 



E 

iS 



, (5) 



where n^g^{= J2crPlsriaPiSv<y) is the number operator, and 775 
indicates the 2p|5| orbital. The first (second) term represents 
the energy difference between the Cu eg and O 2p orbitals, 
which (do not) form the a bond, and A and Ap are the energy 
parameters. The third and fourth terms in Eq. ^ represent 
the on-site electron-electron interactions in the Cu and O sites, 
respectively. These are given by 



n 



Ud 



27 



^dE E 4',A-f<T4','a'dzY<T' 
icTcr' 7>7' 



27' (7 



i(7(7' 7>7' 



E E {41A'ni4YA^l'i + H.c. 



(6) 



i 7>7' 



and 



iSaa' r]>r]' 

— Jp Plsi-iaP'^^Vf^'PiS'n'cr'P'^Wf^ 

i5(7(7' 7]>7]' 

" 4 E E {plsritP'SniPlsri'tP'S^'i + H-c) , (7) 

i6 ri>ri' 

where Ud (Up), U'^ (t/p, Jd ( Jp) and J'^ ( Jp are the intra- 
orbital Coulomb interaction, the inter-orbital Coulomb inter- 
action, the Hund coupling and the pair-hopping interaction for 
the d (p) electrons, respectively. We assume the conditions 
Ud = U'^- 2Jd and = Jd in Eq. ©. 



7 



From the dp-type Hamiltonian, the superexchange interac- 
tions between the NN Cu sites are obtained by the perturba- 
tional expansion. The exchange processes are classified into 
the two processes; two holes occupy virtually the same Cu (O) 
site in the intermediate states, termed the dd (dpd) processes 
as shown in Fig. |5] The superexchange interactions for the 
NN ij sites labeled by / through the dd processes are given by 



n 



dd 



2t 



* J 



-Cd\-^-2T\ 



(8) 



where A, = i!^i\l{^\U',- J,)\. B, = t'^ty[l:.\U', + J^)], 
and Cd = t'^fjj^/iA^iUd + Jd)]- The transfer integrals td and 
t'p are defined by the Slater- Koster parameters as —td = [pda) 
and — = \{ppo') ^ ^(ppTr), respectively. iH The spin- 
singlet and spin-triplet operators are introduced by Pj^ = 



S, ■ S, and 



Si ■ Sj, respectively, where Si is 



the spin operator with an amplitude of 1/2 at site i. The orbital 
degree of freedom in a Cu ion is represented by the pseudo- 
spin operator Ti with an amplitude of 1/2. For convenience, 
we introduce tI = cos{2Tmi/i)T[ ~ sin(27rn,/3)rf and 
fl = cos(27rr7,;/3)Tf + sin(27rn;/3)l^f with {nz,nx,ny) = 
(0, 1, 2). The eigen state of r' with the eigen value of H-1/2 (- 
1/2) corresponds to the state where d^i2_j.2 ((i„i2_„2) orbital 
is occupied by a hole. In the same way, the exchange interac- 
tions through the dpd processes are given by 



n 



y(0 

dpd 



A, (4 - 4r^ - 4rj 
(Bp + 2Cp) 
(4-4t^-4t' 



- 4r'T' 



. 4r'r' 
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Vlflf^P^, (9) 



where = t'Xl[^\K'Jp+'^^)l = t'^tVWU' + 
,/p + 2A)] andCp = i2^4(f7p + 2A- Ap)/[A4{([/p + 2A)^- 
J'p — Ap}]. The transfer integral tp is defined by the Slater- 
Koster parameters as —tp = ^{ppa) + ^(ppir). ^ 

The two-types of the superexchange Hamiltonians are now 
taken together: 



''dd ^ '''dpd 



Hcxch — ^ ^ Y^d 

<ij>l 

^ [jssS, ■ s, + JrrTlr] + j,,fif^ + jy^ypy 



<i3>l 



JssrSi ■ Sjirl + '''j) + JssTrSi ■ SjT^Tj 



'J ssrr*^i 'i ' j ^ ^ ssyy ^J-^i j 



(10) 



where the exchange parameters are defined by Jg, 



}Ad- 



^Bd + 4:Bp + |Crf + 8Cp, Jtt = ^Ad — iAp — B 
J _ 9 /I , I n /I I QD I 3/^, I ar< t 



\Cd — 2Cp, J', 



Ad + 9 + 3Bp + 1 Crf + 6Cp, Jy 



4^a 



f-Brf— |Cd, Js. 



AAp~2Bd-ABp-2Cd~8Cp, 



Jssrr = 5Ad - 4Ap + ABp + 5Cd + 8Cp, J.sff = -3Ad + 
12Ap-UBp^3Cd-24Cp and Jssyy - -3Ad-6Bd + 3Cd. 

We estimate the exchange constants quantitatively. By us- 
ing the realistic parameter values, td ~ 1.2eV, tp = 



t'p = -0.65eV, U'a 



6eV, U„ 



4eV, = 3eV, 



Jd/K = Jp/Up = 0.5, A = 3eV and Ap = O.leV, we 
have Jtt/Jss = 0.78, Jsstt/Jss = 4.5, Jsst/Jss = 1-7, 
Jff/Jss ~ 2.3, Jssff / Jss — ^5.2, Jyy/Jgg ~ — 0.89, and 



J ssyy I J S: 



-3.1 as a unit of J, 



6.5meV. These val- 



ues are adopted to analyze the spin and orbital states in the 
two NN Cu sites. In the numerical calculations for T-L^fi — 
^cxch + T~L,]T in the main article, we adopt the above ratios of 
the exchange constants as a unit of Jss/ J An = 0.15. 



Generalization of the superexchange interaction 

In the main article, we examine the connection between the 
superexchange Hamiltonian in Eq. ( fTot and the honeycomb- 
lattice orbital-only model given by 



(11) 



by generalizing the electron transfer integrals as follows. We 
modify the transfer integral between the p^ and d3^2_,.2 
orbitals and that between p^ and dj.2_y2 along the x di- 

rection as (t^r^' , t^f ^ ^ (-t./2, t,V3/2) ^ 



pd]ix ' pd\ix 

(^prf;^'' '''iv).tpd;ix '""W) = (-id COS?/, id sin?/) by intro- 
ducing a parameter ij. The transfer integrals between other 
orbitals and those along other directions are obtained by the 
symmetry considerations. The generalized transfer integrals 
at 77 = 7r/3 reproduce the original transfer integrals. At 77 = 

n/2, we have fpf-f'^iv = ^/2), ip^-f '^(r? = ^/2)] = 
(0, td), i.e. the electron transfer between the px and ^322.^2 
orbitals along z vanishes. 

Through the generalization of the transfer integrals, we 
derive the superexchange Hamiltonian. The generalized ex- 
change constants are given by Jssiv) = ^^d — 4Ap + 
2Bd + 4Bp + Cd + SCp + i-Ad + 2Bd + Cd) cos^ 2?/, 
Jrriv) = 3Ad-Cd+{3Ad-l2Ap-ABp-Cd-8Cp) cos^ 2i], 
Jffiv) = (-3^d + l2Ap + ABp + Cd + 8Cp)sm^2ri, 
Jyyiv) - -(3^d - 2Bd + Cd) sin^ 2?/, J,,, (77) = 4(2 Ap - 
Bd-2Bp-Cd-ACp)coii2ri,Jssrr{il) = 4(Ad + Cd+(Ad- 
4Ap + 4Bp + Cd + 8Cp)cos2 2?7), Jssftiv) = -4(Ad-4Ap + 
4Bp + Cd + 8Cp) sin^ 2??, and Jssyyiv) = -4(Ad + 2Bd - 
Cd) sin^ 277. At 77 = tt/2, the superexchange Hamiltonian is 
given by 

"^cxch ~ ^ ^ [•^ss'S'j • Sj + Jt-t-T^T^ 

<ij>l 

+ JssrS, ■ Sj{tI + rj) + JssrrS^ ■ 5,r^rj] . (12) 
To reproduce the orbital-only model, we further introduce the 
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staggered magnetic field defined by 



(13) 



In the limit of h ^ oo, the spin degree of freedom is frozen, 
and Eq. (fTSl i is reduced to the orbital-only model in Eq. (fTTI) . 
The exchange constant J corresponds to Jrri'U = 7i'/2) — 
Jgg^^(ri = 7r/2)/4 which is negative. 



and vibrational states, respectively. The adiabatic potentials 
for an octahedron are given as 



fe=±) 



±Ap + Bp^ cos 9, 



(15) 



where 9 = tan^^(Q.u/(5„). The wave functions for a hole 
corresponding to the lower adiabatic potential, i.e. /c = — , is 
given as 



VIBRONIC HAMILTONIAN 

In this section, a detailed derivation of the effective vibronic 
Hamiltonian in Eq. (2) in the main article is presented. 

Derivation of the effective vibronic Hamiltonian 

We start from the electron-lattice interaction Hamiltonian 
introduced in the main article. The two vibrational modes, 
and Qy, with the E symmetry couple with the eg electronic 
orbitals. This is given by 



He, = 



1 



92 



92 



2M \dQl, dQl, 



-Pi 



, (14) 



where pi = \jQ\i + Q\i is an amplitude of the lattice dis- 
tortion at the i-th octahedron. The first two terms are for the 
harmonic vibrations with frequency uj and oxygen mass M. 
The third term describes the Jahn-Teller (JT) coupling with 
a coupling constant A(> 0), and the last term represents the 
anharmonic lattice potential, where B is negative. 

We analyze this Hamiltonian based on the Born- 
Oppenheimer approximation. The vibronic wave-function 
is written as a product of the electronic wave function, 
■i/jfe, and the lattice wave function, (^\, as $A fc(r,<5) — 
i^kir, Q)(I)\{Q), where r and Q are the electron and lattice 
coordinates, respectively, and k and A describe the electronic 



^DJT — 

h,M = 



J, 



DJT 7^ 



1 



t 



t2hf 



'DJT 



t 



'DJT 



:2h, 



-Ef^ 

I 



FIG. 6: The energy level scheme for the vibronic states. 



-0x2_,y2 (r) cos - - ■i/'3z2-r2 (r) sin ; 



(16) 



In the case of B = 0, ^ takes its minimum of Ejt = 
AV(2A/w2) at p = po = A/{Muj^) for any 9. When B is 
taken into account, the potential takes its minima (maxima) at 
angles 6*^;^ = iitt + 2z/7r/3 with integers = (/i = 1) and 
1/ — (0, 1, 2), as shown in Fig. 1(b) in the main article. 

Here we assume that the zero-point vibration energy (w / 2) 
is sufficiently smaller than the JT energy (-Ejt), and the vi- 
bronic motion is confined on the lower adiabatic potential. 
Then, the effective Hamiltonian is given by 



2M \dQl dQl 



X i)-{r, Q) 

}_\ 

^ [ 2M dp^ 
1 92 



E. 



IT 



VP 



2A/p2 d02 



Bp-^ cos 36'. 



(17) 



The vibronic motion is classified into the radial mode where 
the amplitude p varies around pq, and the rotational mode 
where the angle 9 varies at p = po- We consider the case that 
the excitation energy for the radial mode, ui, is larger than the 
kinetic energy for rotational mode, 1 / (2A'/pq) = oj'^ / {AEjt), 
and focus on the rotational mode, corresponding to the last 
two terms in Eq. (T% . The effective Hamiltonian for the rota- 
tional mode is defined as 



Hiot — ^ 



1 



2Mp2 9612 



Bpl cos 36*. 



(18) 



Since this Hamiltonian describes the kinetic motion under 
the periodic potential, solutions of the Schrodinger equation, 
'Hrot4'x{(^) = £\<t>\{(^)^ are obtained as the Bloch states. 
As shown in Fig. |6l the lowest-six eigen states are labeled 
by the irreducible representation in the Csv group, E'^^\ Ai, 
A2 and E'-"\ where E'-^^ and E'-"'^ are the doubly degener- 
ate representations and the degenerate bases are labeled by 
the indexes + and — . From the Bloch-type eigen states, 
we introduce the Wannier-type wave functions. From the 
lowest-six eigen states, we introduce the six Wannier func- 
tions, (t)Qu{9) and 01^(6') {v ~ 0, 1, 2), which are almost lo- 
calized around 9qi, and 9ii, respectively. Explicit relations 
between the Bloch functions and the Wannier functions are 



given by 



„±2i7r 
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(j^At = {4>m + </'oi + 002) /a/3, ^A; 
and (h^(H) 



^'io + <^ii + 0i2)/y3 



e^^^'^/^^ii +e±4"/30^2)/A/3. Because 



of the parity of the wave functions, the two spaces based on 
the wave functions 4'qi,{6) and 4'ii,{9), termed the /i=0 and 1 
spaces, respectively, are orthogonal with each other and are 
treated independently. 

The vibronic wave-function is represented by a product 
form of ^fj_,y{r;d) ~ '4}-{r\9)4)^^{9). For simplicity, we 
assume that the energy difference between the ij(^) and Ai 
states is equal to that between A2 and E^-^\ We denote this 
energy difference by Adjt and that between the Ai and A2 
states by A ah- Then, the low-energy vibronic Hamiltonian is 
obtained as a simple form: 



A 



DJT + j 



X -1$ 



.(i) I 



(19) 



This is rewritten in the Wannier function representation as 



n 



JT 



(20) 



where the 2x2 matrix af is the z component of the Pauli 
matrices, and is defined for the index ji. We define Jdjt = 
2Adjt/3 and Jah = 4Adjt/3 + Aah, by which the condi- 
tion Jdjt/^ah < 1/2 is derived. 

We introduce the 3x3 matrices {D^)^^i = |eiiyi/'|' where 
ei^^i are the Levi-Civita completely antisymmetric tensors, 
for the index v. By using the matrices, we have 



^iii^ ifJ- = 0,1; = 0,1,2), are adopted as a basis set. 
This is performed by introducing the projection operator as 
Hcxch VHcxchP. Here V operates the vibronic wave func- 
tion so as to be restricted within the six basis functions (1>^^ 
by changing a value of 9. 

Artificial field for vibronic state 



In the main article, we examine a connection between the 
Hamiltonian Hcs = "Hoxch + ^jt and the Heisenberg model 
Hspin = J '12 (a) '^i ' '^j by introducing the artificial field for 



the orbital-lattice sector We introduce the external field which 
acts on the eigen states 



,(1.) and ^„{H) as 



n 



M 



-|*b(-))(*b(-)| + I^k(->)(*b(-)| • (23) 



In the Wannier function representation, we have 



(24) 



where the 2x2 matrix, erf, is given for the index /i. The 3x3 
matrix is defined by (F)^,,/ = J^i ^ii^i^' for the index 
and is explicitly given as 



F, 




(25) 
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1 






E^' = 


1 







(21) 
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(22) 


i=0 


V 










On an equal footing of the effective vibronic Hamiltonian 
in Eq. (|20] |. we provide a representation for the superex- 
change Hamiltonian, where the six vibronic wave functions. 



A schematic energy levels under the external field is given 
in Fig.|6] Double degeneracies in the £''^) and i?'^' levels are 
lifted by applying the virtual field. In the limit of Km — > 00, 
the orbital and vibrational degrees of freedom are frozen, and 
'Heff = 'Hexch + 'Hdjt IS rcduced into the Heisenberg model 
^spin on a honeycomb lattice. 
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